Add benchmarking with CI job - #9

Hey @vvish thanks for the PR. I'm also looking into adding benchmarks in #8 where I also refactor to share some factories between test and benchmarking.

There I have added a map_unary (yet do do binary / n-ary) that significantly overlap with transform.
The idea is for it to be more generic and free from the STL apis constraints (leverage alignment knowledge, purity, also no macros), handle more complex functions (e.g. xsimd::widen) and different type widths.
It seems to perform a bit better on small and medium runs. Ideally transform would become a special case of it.
Curious to know what you think.

As for benchmarking in CI, the setup suggested here will be extremely noisy. We plan to add the same approach as xtensor with namespace+codspeed.

Hey @vvish thanks for the PR. I'm also looking into adding benchmarks in #8 where I also refactor to share some factories between test and benchmarking.

There I have added a map_unary (yet do do binary / n-ary) that significantly overlap with transform. The idea is for it to be more generic and free from the STL apis constraints (leverage alignment knowledge, purity, also no macros), handle more complex functions (e.g. xsimd::widen) and different type widths. It seems to perform a bit better on small and medium runs. Ideally transform would become a special case of it. Curious to know what you think.

As for benchmarking in CI, the setup suggested here will be extremely noisy. We plan to add the same approach as xtensor with namespace+codspeed.

Hello @AntoinePrv ,

Thank you for the reply.

You are right - the variance on GH Action runners is huge. That is why I abandoned the initial idea to store the results and keep the baseline to track changes. The benchmarks in the PR compare xsimd-based implementations with std-based to provide relative speedup that I think is necessary as well. CI job is just the simplest way to execute it for various target platforms. If you plan to integrate codspeed - it is great and resolves the variance issue.

So the question: in your PR you are implementing benchmarks for the map_* functions. How benchmarks can be introduced for the rest? Also I would suggest to keep relative benchmarks as well as they provide guidance for developers and users.